tests: Scripts for e2e tests #2128
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
|
The PR also broke the unit tests. I'll send a fix for them. We should be able to run unit tests for GH actions though, since this is not resource intensive. /assign @kimwnasptd |
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
kimwnasptd
force-pushed
the
feature-kimwnasptd-e2e-tests
branch
from
7391b01
to
6ee37ce
Compare
|
After inspecting the VM sizes of the GH actions I realized that the machines were running out of memory when I was installing all the manifests. It's a 7Gb VM. I managed to created a "light" version of the manifests that only install the barebone components, to get the tests running. This time it took 6mins to install, but in the end got timeouts during training. This is most probably due to CPU limitations. @StefanoFioravanzo @elikatsis I'm going to remove the GH action and fix unit tests. Then as a next step I'll implement this by using Prow. |
To avoid having to use --load_restrictor none we'll need to wrap the KServe manifests inside a kustomization.yaml file. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
We should use prow instead to trigger our e2e tests. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
|
@elikatsis @StefanoFioravanzo this should be ready for review. To summarize:
The next step will be to use Prow, with workflows similar to what we do for Notebooks, in order to run our e2e tests there. https://github.com/kubeflow/kubeflow/blob/master/prow_config.yaml |
|
Thanks for all the effort and the troubleshooting @kimwnasptd. Orchestrating all of this testing.. phew.. That's great! /lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: elikatsis, kimwnasptd The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
kimwnasptd
added a commit
to arrikto/kubeflow-manifests
that referenced
this pull request
Feb 15, 2022
* remove old test files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gitignore: Don't track pyc files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * flake8: Introduce linting file Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * hack: Introduce scripts for cluster manipulation Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * tests: Add e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * GH action for running e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Reduce the installed components and system reqs Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Add simple kustomization file To avoid having to use --load_restrictor none we'll need to wrap the KServe manifests inside a kustomization.yaml file. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * unittests: Fix unit tests Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gh: Remove action for e2e tests We should use prow instead to trigger our e2e tests. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
google-oss-prow bot
pushed a commit
that referenced
this pull request
Feb 16, 2022
* tests: Scripts for e2e tests (#2128) * remove old test files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gitignore: Don't track pyc files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * flake8: Introduce linting file Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * hack: Introduce scripts for cluster manipulation Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * tests: Add e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * GH action for running e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Reduce the installed components and system reqs Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Add simple kustomization file To avoid having to use --load_restrictor none we'll need to wrap the KServe manifests inside a kustomization.yaml file. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * unittests: Fix unit tests Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gh: Remove action for e2e tests We should use prow instead to trigger our e2e tests. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add networkpolicies under /contrib/networkpolicies (#2121) * Create .gitkeep * Add files via upload * Create OWNERS * Create README.md * Delete default-deny-not-istio-system.yaml * Create default-allow-same-namespace.yaml * Create centraldashboard.yaml * Create jupyter-web-app.yaml * Create katib-ui.yaml * Create kfserving-models-web-app.yaml * Create ml-pipeline-ui.yaml * Update ml-pipeline.yaml * Create volumes-web-app.yaml * Update kustomization.yaml * Update OWNERS * Sync kubeflow pipelines manifests 1.8.0 rc.2 (#2131) * hack: Update pipelines sync script to change README Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/pipelines manifests from 1.8.0-rc.2 * Sync kubeflow kubeflow manifests v1.5.0 rc.1 (#2134) * hack: Sync README for kubeflow/kubeflow sync-script Extend the sync-script for kubeflow/kubeflow to also update the components versions in the readme. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/kubeflow manifests from v1.5.0-rc.1 * Sync kserve/models-web-app manifests (#2135) * kserve: Rename from upstream to kserve We will be including both kserve/kserve and kserve/models-web-app into the manifests, so the names will need to reflect this. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Add manifests for the models-web-app Include the MWA manifests from the v0.7.0 tag. https://github.com/kserve/models-web-app/tree/v0.7.0 Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Include both kserve and mwa manifests Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/kfp-tekton manifests from v1.1.1 (#2141) * hack: Update tekton script to edit README The hack script for updating the kfp-tekton manifests should also be updating the README file as well. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/kfp-tekton manifests from v1.1.1 * Update manifests for Katib v0.13.0-rc.1 release (#2139) * Update manifests for Katib v0.13.0-rc.1 release * Change README * readme: Remove MPI reference and add ingress distributions link (#2143) * Closes #1963 * Remove unused MPI reference (PR #2119) * Update kubeflow/pipelines manifests from 1.8.0 (#2144) Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * hack: Don't error if namespace kubeflow exists (#2140) The helper setup scripts should not error when the namespaces already exist. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> Co-authored-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Co-authored-by: a9p <58503488+a9p@users.noreply.github.com>
VaishnaviHire
pushed a commit
to VaishnaviHire/manifests
that referenced
this pull request
Aug 11, 2022
* tests: Scripts for e2e tests (kubeflow#2128) * remove old test files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gitignore: Don't track pyc files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * flake8: Introduce linting file Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * hack: Introduce scripts for cluster manipulation Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * tests: Add e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * GH action for running e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Reduce the installed components and system reqs Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Add simple kustomization file To avoid having to use --load_restrictor none we'll need to wrap the KServe manifests inside a kustomization.yaml file. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * unittests: Fix unit tests Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gh: Remove action for e2e tests We should use prow instead to trigger our e2e tests. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add networkpolicies under /contrib/networkpolicies (kubeflow#2121) * Create .gitkeep * Add files via upload * Create OWNERS * Create README.md * Delete default-deny-not-istio-system.yaml * Create default-allow-same-namespace.yaml * Create centraldashboard.yaml * Create jupyter-web-app.yaml * Create katib-ui.yaml * Create kfserving-models-web-app.yaml * Create ml-pipeline-ui.yaml * Update ml-pipeline.yaml * Create volumes-web-app.yaml * Update kustomization.yaml * Update OWNERS * Sync kubeflow pipelines manifests 1.8.0 rc.2 (kubeflow#2131) * hack: Update pipelines sync script to change README Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/pipelines manifests from 1.8.0-rc.2 * Sync kubeflow kubeflow manifests v1.5.0 rc.1 (kubeflow#2134) * hack: Sync README for kubeflow/kubeflow sync-script Extend the sync-script for kubeflow/kubeflow to also update the components versions in the readme. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/kubeflow manifests from v1.5.0-rc.1 * Sync kserve/models-web-app manifests (kubeflow#2135) * kserve: Rename from upstream to kserve We will be including both kserve/kserve and kserve/models-web-app into the manifests, so the names will need to reflect this. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Add manifests for the models-web-app Include the MWA manifests from the v0.7.0 tag. https://github.com/kserve/models-web-app/tree/v0.7.0 Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Include both kserve and mwa manifests Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/kfp-tekton manifests from v1.1.1 (kubeflow#2141) * hack: Update tekton script to edit README The hack script for updating the kfp-tekton manifests should also be updating the README file as well. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Update kubeflow/kfp-tekton manifests from v1.1.1 * Update manifests for Katib v0.13.0-rc.1 release (kubeflow#2139) * Update manifests for Katib v0.13.0-rc.1 release * Change README * readme: Remove MPI reference and add ingress distributions link (kubeflow#2143) * Closes kubeflow#1963 * Remove unused MPI reference (PR kubeflow#2119) * Update kubeflow/pipelines manifests from 1.8.0 (kubeflow#2144) Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * hack: Don't error if namespace kubeflow exists (kubeflow#2140) The helper setup scripts should not error when the namespaces already exist. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> Co-authored-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Co-authored-by: a9p <58503488+a9p@users.noreply.github.com>
kevin85421
pushed a commit
to juliusvonkohout/manifests
that referenced
this pull request
Feb 28, 2023
* remove old test files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gitignore: Don't track pyc files Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * flake8: Introduce linting file Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * hack: Introduce scripts for cluster manipulation Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * tests: Add e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * GH action for running e2e test Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Reduce the installed components and system reqs Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * kserve: Add simple kustomization file To avoid having to use --load_restrictor none we'll need to wrap the KServe manifests inside a kustomization.yaml file. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * unittests: Fix unit tests Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * gh: Remove action for e2e tests We should use prow instead to trigger our e2e tests. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a first step for #2099
The PR reworks the entire
testsrepo to include a simple e2e test that:While developing the tests locally I was using KinD to spin up clusters and test everything from scratch. With this mentality, and along the fact that we don't have many maintainers in the
kubeflow/testingrepo, I tried to use GH actions to run these E2E tests.But the performance was abysmal. Even with optimizations, like loading etcd in memory kubernetes-sigs/kind#845, I couldn't get the tests to succeed. It took more than 45 mins to apply all the manifests and then the test failed due to etcd timeouts.
I've included the GH workflow files, to give it one last try, but I don't have many expectations. The next step for this is to see how could we use the AWS infra/prow to spin up the clusters and try our luck with there. The current blocking PR for this is kubeflow/testing#972.
cc @kubeflow/release-team
/cc @StefanoFioravanzo @elikatsis